Skip to content

docs(operator-guide): webhook certificate rotation and key compromise - #655

Merged
MaxRink merged 3 commits into
mainfrom
docs/webhook-cert-rotation
Nov 15, 2026
Merged

MaxRink merged 3 commits into
mainfrom
docs/webhook-cert-rotation

Conversation

@MaxRink

@MaxRink MaxRink commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Closes #650

Description

Adds a "Webhook Certificate Rotation" section to docs/operator-guide.md (under Security Considerations) and links it from the chart README. The section covers:

  • the Secret <release>-webhook-certs, its four keys, and who can read it with the chart defaults. This includes the controller-manager ServiceAccount, whose ClusterRole grants get/list/watch on *.
  • the cert-controller defaults: CA valid for 10 years, serving certificate for 1 year, renewal 90 days before expiry, a check every 12 hours and on every Secret change.
  • that the rotator injects caBundle into every configuration passed with --cert-rotation-*-webhook. That is the binder webhook, plus the namespace webhooks and the pod deletion guard when they are enabled.
  • that the /authorize kubeconfig on the API servers is not updated by the rotator.
  • inspection commands: certificate dates and fingerprint, a stale-caBundle check across all webhook configurations of the release, and an openssl s_client check of the served chain.
  • a planned rotation: remove .data, restart the deployment, verify. It also describes the admission outage window caused by failurePolicy: Fail.
  • the steps after a suspected compromise: close the access path, rotate, rotate the /authorize token, and check the audit logs for Secret reads.

The defaults match t-caas-go-library pkg/certrotation v0.1.5, which uses cert-controller v0.16.0 defaults. telekom/t-caas-go-library#52 makes them configurable and does not change the defaults.

Relation to #642

Draft #642 (external CA through chart values) is still open. This PR documents the current main behaviour and does not conflict with it: #642 adds its own "Providing the Webhook CA" section, including how to replace a provided CA. Once #642 merges, the "Planned Rotation" step "remove .data" does not apply to a provided CA, because removing it makes the rotator generate a CA that the API servers do not trust. A follow-up should add a pointer to #642's section at that point.

Type of Change

  • 📚 Documentation update

Testing

Every command in the section was run verbatim (extracted from the Markdown) against a kind cluster. The cluster ran the Helm chart from this branch with image 0.5.0-rc.26 (equal to main), 2 webhook replicas and namespaceAdmission.enabled=true:

  • the inspection commands report all three webhook configurations as ok. Right after the Secret was emptied, the loop reports all three as STALE until the new leader injected the CA, about 20 to 30 seconds later.
  • after the rotation the CA fingerprint changed. openssl s_client verifies the served chain against the new ca.crt, and a server-side dry-run of a BindDefinition and a Namespace passes the webhooks.
  • kubectl auth can-i confirms Secret access for the webhook-server and controller-manager ServiceAccounts.

The audit-log step is prose only, because audit backends differ.

PSA PSM-13.10

🤖 Generated with Claude Code

Document the webhook certificate Secret, who can read it, the
cert-controller defaults, how to inspect and verify the CA injection,
a planned rotation and the steps after a suspected key compromise.
Link the section from the chart README.

Closes #650

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: MaxRink <maximilian.rink@telekom.de>
@MaxRink
MaxRink requested a review from a team as a code owner November 15, 2026 11:06
Copilot AI balanced review requested due to automatic review settings November 15, 2026 11:06
@github-actions github-actions Bot added documentation Improvements or additions to documentation helm labels Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Output Delta

No output differences were detected for this PR.
No output-delta input files changed.
There is no diff to inspect for this run.

  • Trusted baseline job: skipped
  • Candidate PR job: skipped
  • Differences detected: false
  • RBAC differences detected: false
  • CRD status differences detected: false
  • Candidate log errors detected: false
  • Latest tag diff generated: false
  • Workflow conclusion: skipped
  • Workflow run: not_applicable

The compare job downloads artifacts only; it does not checkout or execute pull-request code.

@github-actions

Copy link
Copy Markdown
Contributor

📊 Output Delta Report

No output differences were detected for this PR.
No output-delta input files changed, so the Output Delta comparison workflow was not run.

No full diff content was generated for this run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The operator guide contains incorrect resource names, a verification check that can falsely report success, incomplete certificate and token rotation instructions, and audit guidance that excludes potentially compromised operator identities.

5 open issues
What changed in this PR

Adds operational guidance for managing Auth Operator’s webhook certificates, covering routine rotation and suspected key compromise.

Changes:

  • Documents certificate storage, Secret access, rotation defaults, and certificate verification commands.
  • Adds planned rotation and compromise-response procedures, including admission outage warnings and separate /authorize configuration updates.
  • Links the new guidance from the Helm chart README.
File Description
docs/​operator-guide.md Adds certificate inspection, rotation, and key-compromise guidance under Security Considerations.
chart/​auth-operator/​README.md Links the certificate auto-rotation note to the new operator-guide section.

Review effort: Balanced

Comment thread docs/operator-guide.md Resolved
Comment thread docs/operator-guide.md Resolved
Comment thread docs/operator-guide.md Resolved
Comment thread docs/operator-guide.md Resolved
Comment thread docs/operator-guide.md Resolved
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Security scan report

Commit: 4ad7c34171f0e2380627c209022bb47c29fcc3c2

Security findings are advisory, not merge blockers. Build, test and licensing checks remain blocking.

A successful job is not proof of a clean scan; findings and scanner errors are called out below.

Check Result Details
Trivy Image Scan success Logs
Helm Chart Security Scan success Logs
Trivy Filesystem Scan success Logs
Go Vulnerability Check success Logs
Dependency Review success Logs
Go Security Checker success Logs
Lint success Logs
Dependency License Audit success Logs
Helm Lint success Logs
Verify Release Image Signature skipped Logs
Build success Logs
Security Scan success Logs
Test success Logs
Verify Generated Files success Logs

See the linked logs for complete findings and any scanner setup failures.

MaxRink and others added 2 commits November 15, 2026 12:27
Use the chart full name instead of the release name for object names,
report unreadable webhook configurations and an empty CA instead of ok,
describe both admission failure windows of a rotation, update the API
server kubeconfig when rotating the /authorize token, and keep the
operator ServiceAccounts in the audit search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: MaxRink <maximilian.rink@telekom.de>
The caBundle check selects webhook configurations by the Helm instance
label, which Kustomize installs do not set, so the loop printed nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: MaxRink <maximilian.rink@telekom.de>
@MaxRink
MaxRink merged commit 02526e2 into main Nov 15, 2026
54 checks passed
@MaxRink
MaxRink deleted the docs/webhook-cert-rotation branch November 15, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation helm size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(operator-guide): webhook certificate rotation and key compromise

2 participants